No-Code AI Avatar that Remembers Your Notes and Answers 
Questions with its Own Voice using GPT-3, Siri and 
ElevenLabs 


Power of Al & no-code Automation 


Avatar 


Both ChatGPT and GPT-3 are great tools if used wisely. By writing good 
prompts, you can unlock their true potential. But there is something more, 
which is an access to the Internet and knowledge about YOU! And the fun part 
is, that you can achieve this with no-code tools like Airtable, Make.com and 
Pinecone (vector database). 


In a previous post, Ishared my vision of an Al Assistant. In my other post, you 
can find a detailed implementation in Node.js. This time, I will show you how to 
do the same with a single line of code (rly!). 


Required Apps & Services 


Our AI Avatar needs to have a memory. In our case, we will use Airtable; its API 
and UI make it simple and easy to use. In a database we need two tables: 
Memory (for storing long-term data) and Conversation (for storing a current 
conversation). 
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Note: At some point, there will be a need to use the premium plan of Airtable. 


While storing information is crucial, there is something even more important: 
searching. In theory, we could use a search engine such as Algolia or 
ElasticSearch, but configuration would be much more difficult and would 
require some coding. To make it really easy and effective, we'll gonna use a 
Vector Database called Pinecone. It will perform something called a "similarity 
search". So, every time we ask Avatar a question, Pinecone will search for 
meaningful information, based on which we will get a response. 


Do not worry! For now, there is like nothing more you need to know about this. 
We just need a three, easy API calls to get through this. I'll show you that in a 
minute. 


Note: Pinecone has a good free plan for learning & small projects. At some point 
there may be a need to use its paid version. 


A third app is Make.com. If you don't yet know about it, #withMake you can 
easily connect to hundreds of services and even any custom API. You need a 
minimum Core or Pro plan of Make to get started. Avatar's main logic will be 
built in a scenario such as this: 


Mind 


I assume that you have at least basic experience with make, but I'll try to stay as 
detailed as it possible. 


And the last one is GPT-3 API, which is a paid service! You should definitely set 
a hard limit and you're using those tools at your own risk. Regardless of 
whether you use Node.js or Make.com, there is a chance of making too many 
API calls for which you will have to pay real money. Keep that in mind, because 
a hard limit is our only hope (ha. ha.). 


To put all of this together, overall flow looks like this: 


make.com is logic 

. Pinecone is smartness 

. Airtable is memory 

GPT-3 is mouth 

. (bonus) ElevenLabs is voice 
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When we speak with Avatar it will: 


. Connect with a make.com scenario 

. Fetch relevant information with Pinecone 

. Get actual data from Airtable 

. Generate a response with GPT-3 

. And return it with make (optionally, it may be sent to EL to convert text to 
speak) 


ok wù Ne 


Long term & Short term memory 


Let's deep dive into Airtable database. As you can see on a screenshots above, 
we have a couple of columns and views. 


Memory table 
In the memory table, we have two views: "All" and "To Sync". The first one 


includes all of the records we have, and the second filters those in which the 
"Sync" column is checked: 
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Memory table includes columns: 


e record_id (type formula with a value: RECORD_IDỌ0), 

* content (Long text), 

e source (Single line), 

e Sync (Checkbox), 

e Forget (Button that will redirect us to the webhook. We'll back to this later) 


In this table, we'll store all of the information that our Avatar will have access to. 


Conversation table 


Second table is really simple: 


e record_id (type formula with a value: RECORD _IDO), 

question (Long text), 

e answer (Long text), 

e summary (Long text), 

* minutes (Formula: DATETIME _DIFF(CREATED_TIME(Q, NOWO, "minutes")) 
e created_at (Created Time) 


In those columns, we'll store a current conversation and summaries. 


Smart Search (Similarity Search) 


The next step is to setup a database in Pinecone and this is really simple. Create 
a free account and use settings like this: 
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Indexes in the Starter Plan will be terminated after 14 days of inactivity 


CREATE INDEX CANCEL 


Hit "Create Index" and then copy two values: the index URL and your API key. 
Save them to your notes, as we'll need them later. 


Index Name 


monday 


monday- sone.io 
@ Ready 


Believe this or not, but now you can close Pinecone, because its API is all we 
need. 


Initialize Logic 


It's time to create a new scenario in make.com. It needs to be triggered with a 
Webhook, which will be listening for HTTP POST requests with a JSON body. 


First three modules like this: 


Webhooks OpenAl Key Pinecone Key 


Custom webhook Set variable Set variable 


Webhook will accept POST HTTP requests with a JSON body like this: 


"content": "[query]", 
"record_id": "recXXXXXXX" 
"type": "ask" // forget / save / remember 


Content is a query from us; it may be information to be saved or answered to, 
depending on the type property. The record_id will be used only for Airtable 
interactions, and the type property describes which action a scenario needs to 
take. Everything will be clear in a moment. 


Two additional modules are actions "Set Variable" from "Tools" module (it's 
native make's module). As values of those variable, provide both names like 
openai_key and pinecone_key and then paste your API keys from OpenAI and 
Pinecone. 


Storing information in Airtable 


There is a difference between storing (in Airtable) and memorizing (in 
Pinecone). Because of that, we will split this into two steps. 

Storing information will require us to modify our scenario by adding a "route" 
like so: 


F Save record 


Create a Record 


Webhooks Router 


Custom webhook 


OpenAl Key 


Set variable 


As you can see, between Webhooks and OpenAI Key, I have added a Router 
module and connected it to both the Create a Record action from Airtable and 
the rest of the scenario. 


There are some important details here: filters. If you click on chains between 
modules, you'll see a "Filter" option. First one is this between Webhook and 
Router and looks like this: 
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x 
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As you can see I'm checking here if there is a content or record_id variable 
available in a payload. 


A second and third filters checks if type is set to "save" or to some other value: 


Router 


=) 
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First route: Type = save 


The fallback route. It will be used in the case where a bundle 
cannot continue on from the router via any other route. For 
more information, please see the online Help. 


Yes © No 


Condition 


x 


Text operators: Not equal to z 


Router 


save 


Add AND rule Add OR rule 
| Cancel 


Second route: Type != save 


It's really important to make sure that those filters are set properly. 


If a type is set to "save", we need to store informations in Airtable like below. 
Just double check if sync column is set to "Yes" (you can ignore blurred part 
from a screenshot below). 


Connection 
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9 For more information on how to create a connection to Airtable, 
see the online Help. 
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Now you can use any HTTP client to make a POST request to your Webhook. 
Just copy its address. Then save a whole make.com scenario and activate it. 


Webhooks 


© Webhook 
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~~ For more information on how to create a hook in Webhooks, 
see the online Help. 


Show advanced settings 


Webhooks 


Custom webhook 


If you send a JSON with properties content and type (save), an information 
should appear in Airtable. 
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Postman HTTP Request 


Airtable Automation 


Any information that our Avatar needs to remember, will be first saved in 
Airtable with "sync" checkbox selected. Because of this, this record will go into 
"To Sync" view, we have created. Such event may be a trigger for Airtable's 
built-in automations. 


I have created an automation which triggers a JavaScript snippet (as 
mentioned, it's just a single line of code!): 


// Replace WEBHOOK_ID with your own 
fetch( https://hook.eu1.make.com/WEBHOOK_ID? 
type=remember&record_id=${input.config() .idt*) 
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All you need to do is to make sure that automation is connected with "To Sync" 
view, "id" variable is connected with "record_id" field and it is active. 


An idea here is a very simple one. Every time a record will appear in To Sync 
view, it will trigger a JavaScript snippet, which will send an identifier of a record 
to the scenario in make.com. 


If you trigger HTTP request in Postman once again, you'll see that your scenario 
in make.com will be triggered twice: first because of a request you've just sent 
and a second one because of Airtable automation. 


What's more, if you need to update information already saved in Airtable, you 


can just select "sync" checkbox and wait a couple of seconds. At this point 
nothing else will happen, because we don't have a proper logic yet. 


Airtable & Pinecone Sync 


We're ready to sync records with Pinecone. Right after the last "Set Variable" 
module, we'll need additional modules. 
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Let's explain each of them. 


1. Router for splitting the path and filter to verify if record_id variable is 
present in JSON from Webhook. 
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1. We need fetch additional details of this record from Airtable. We can do this 
with action Get a Record from Airtable module. 


Airtable 3 (Oe 


Connection 
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© For more information on how to create a connection to Airtable, 
see the online Help. 
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1. In the next step, we need to create a JSON in the JSON module. To do this, 
first you have to "Add" a Data Structure, which you can generate by pasting a 
JSON object like this: {"input":"", "model":""%. Then, add the content 
field from Airtable and set the Model to text-embedding-ada-002. 


JSON X ? x 
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1. Created JSON needs to be passed with an HTTP request to OpenAl, because 
we need to convert the information from Airtable to something called 
"embeddings". 
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1. We need to send generated embeddings to the Pinecone, but first we have to 
create another JSON object. You can generate a data structure with this 
example: {"vectors":[{"id":"","values":[123]%]%. And fill variables as 
shown on a screen below. Keep in mind that id in Pinecone is same as in 
Airtable. 
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. At this step you need to send information to the Pinecone with HTTP 
module. URL should be set to: 


https://YOUR_PINECONE_URL/vectors/upsert. Also make sure to include 
headers (Api-Key and Accept) and JSON String as a body. 
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Vectors JSON Pinecone: Upsert 


Create JSON Make a request 
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1. Everything is done, so we have to uncheck the checkbox in Airtable to make 
the row disappear from the "To Sync" table. 
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1. And the last step will be a Webhook response that look like this: 


Webhooks : 2 oa 


© Status 
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{"status": "success", "message": "Updated."} 
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At this point, our Avatar can remember any text-information that appears in the 
"To Sync" table in Airtable. 


I don't know if you can see the whole logic here, but what we do is store data 
both in Airtable (plain text) and Pinecone (embeddings). 

Forgetting 

Before we move on to the last step, we need to create a logic for forgetting 


information. It will be triggered manually with a button "Forget" in Airtable. 
Edit it's type like so: 


Forget 
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Make sure to add your webhook address. After hitting this button, our scenario 
will be triggered with type=forget and record_id. 


To our latest Router, add another path and make HTTP Request to the Pinecone 


like on image below. Make sure to include a filter to check if type=forget: 
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That's all. To delete and forget a given information, just hit the "Forget" button 


in Airtable, right next to it. 


Context 


We've got everything we need to create last part of Avatar's logic. To write a 
response, our scenario has to: 


* convert current message (content) to embeddings 

e use those embeddings to query Pinecone database 

e use IDs retrieved from Pinecone to query Airtable 

e get current conversation 

e generate a response 

e send response 

e save current conversation 

e Converting the current message to embeddings and querying Pinecone will 
look the same as it did in a previous path. 


Embedding JSON OpenAl: Create Embedding Query JSON Pinecone: Query 


Create JSON Make a request Create JSON Make a request 


Embedding & Querying 


To convert content to embeddings, pass it as input to the JSON and create 
embeddings, exactly as before. 
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© Data structure that describes the final string. 
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To query Pinecone we need a new JSON. You can create its structure like so 
based on this example: {"topK":7,"filter":4%,"vector": 
[123], "includeMetadata":true}?. 
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I Data structure that describes the final string. 


Filter 
TopK 
Vector EE map F 
Data. data[ 
Include Metadata Map 
Query JSON a R 
t 
Create JSON E j aa 


Show advanced settings 


Querying 


Request to the Pinecone needs to be sent to 
https: //YOUR_PINECONE_URL/query URL with settings like this: 
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1. Now we can use IDs returned from Pinecone to fetch Airtable records. 
Pinecone returned an array, so we have to use "Iterator" module to iterate 
through all of the elements and gather IDs. 


[ Dee A 
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Iterate IDs ad 
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~ Under the Group by field, you can define an expression 
containing one or more mapped items. The aggregated data will 
be then separated under Groups with the same expression's 
value. Each Group outputs as a separate bundle containing a Key 
with the evaluated expression and the aggregated text. By doing 
this, you can use the Key as a filter in subsequent modules. 


© Stop processing after an empty aggregation 


Yes 


Text 


metadata: id 


EE Show advanced settings 


1. With Airtable Formulas, we have to select only those records whose IDs are 


on a list from Pinecone. 


© Connection 


Je E 


© For more information on how to create a connection to Airtable, 
see the online Help. 
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IF(SEARCH ({record_id}, "ESZ "), 1, 0) 
© An Airtable formula used to filter records. The formula will be 
evaluated for each record, and if the result is not 0, false, ", NaN, [, 
or #Error! the record will be included in the response. You can find 
more information about the formula on Airtable's website. 

For example, to only include records where Name isn't empty, 
pass in: NOT({Name} =") 


© Limit 
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1. All content fields have to be merged into a single string. We can do this by 
using the Text Aggregator again. Row separator needs to be set to "New row" 
but also, we need to add more detailed separator like "###" or "\n\n\n". 
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1. The last step of preparing our context will be fetching the last conversation. 
It will be empty at the start, but that's okay. Just make sure you limit the 
results to a single row; it's crucial, as it will make the rest of the scenario 
execute multiple times. 


Ñ WARNING: Read a description above once again and really make sure that a 
limit is set to 1. 
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Answering 


We have all of the necessary information to generate a response. But first we 
have to create a prompt that will describe our Avatar's behaviour. Template of 
this prompt look like this. 


[Avatar behaviour] 
Current date: "{{formatDate(now; "dddd YYYY-MM-DD HH:mm") ¢¢" 


Hitt 

Your memory: 

$$32.texti? 

Hitt 

Conversation summary:{4$if(41.summary; 41.summary; "[no history]")%? 
Hitt 


- [Name] :441.content¢$ 


- Answer: 


Avatar behavior can be defined however you want, but first take a closer look at 


the template above. We have a very clear separation of "Description," "Memory' 
"Summary," and, lastly, the Question and Answer. 


An example prompt may look like this: Keep in mind that the prompt needs to 
be as concise and detailed as possible; every word counts, because even at 
small scale it may generate noticeable costs. 


As Avatar AI conversing directly with your friend, [name]. Respond to 
his latest message based on your memory, current conversation and 
knowledge about yourself, using complex, grammatically correct 
sentences, a natural tone, and if you cannot relate to the question 
or it is not included in your memory, be honest and write "I don't 
know", asking a probing question. Draw conclusions and summarize the 
conversation to find out and understand what you are talking about. 
Pay special attention to the differences between your memory and 
current conversation. Be specific, polite, clear, honest and 
interested. 

HHH 

Your memory: "[name] is a programmer" 

HHH 

Conversation summary: "We spoke about [name]'s dog" 

HHH 

[name]: Can you summarize our current conversation? 


You: 


To test your prompt, you can use OpenAl Playground: 


#8 again to Hide Sidebar 


| Edit the shortcut 7 


@ Overview Documentation Examples Playground 


Playground Load a pre Viewcode Share 


As Avatar AI conversing directly with your friend, Adam. Respond to his latest message based on your memory, 
current conversation and knowledge about yourself, using complex, grammatically correct sentences, a natural tone, 
and if you cannot relate to the question or it is not included in your memory, be honest and write "I don't know", asking Model 
a probing question. Draw conclusions and summarize the conversation to find out and understand what you are 

talking about. Pay special attention to the differences between your memory and current conversation. Be specific, text-davinci-003 
polite, clear, honest and interested. 

HHH Temperature 
Your memory: "Adam is a programmer" 

HH 

Conversation summary: "We spoke about Adam's dog" 
HEE 

Adam: Can you summarize our current conversation? Stop sequences 


Enter sequence and press Tab 


Maximum length 


You: 


Top P 

Frequency penalty 
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Best of 

Inject start text 


@ Looking for ChatGPT? Try it now Z 
Inject restart text 


v 
NE a 


Show probabilities 


We can go back to our scenario. With native GPT-3 module, we can access 
OpenAI API and insert our prompt: 


Model Map 


text-davinci-003 X 


© ID of the model to use. 


© Prompt 


your memory, be honest and write "I don't kno 
w", asking a probing question. Draw conclusions 
and summarize the conversation to find out and 
understand what you are talking about. Pay spec 
ial attention to the differences between your me 
mory and current conversation. Be specific, poli 
te, clear, honest and interested. 


HHH 
Your memory: 


HHH 
Conversation summary: if( ; 


; [no history] ) 
HHH 


- overment: EEB 
- You: © ( 
9 


3) The prompt to generate completions for. I 


Max Tokens 


750 


Get Conversation Answer © The maximum number of tokens to generate in the completion. 


Defaults to 16. 
Search Records Create a Completion 


Temperature 


07 


O What sampling temperature to use. Higher values means the 


model will take more risks. Try 0.9 for more creative applications, 
and 0 (argmax sampling) for ones with a well-defined answer. 
Defaults to 1. 


Based on a response from GPT-3 module, we need to create a JSON using this 
example to generate Data Structure: {"status": "success", "message": 


EEE, 


© Data structure C Refresh 


ol Avatar Response 


© Data structure that describes the final string. 


Add 


© Stat 
Q d © Status e © 
success # 
Message 


tri 


Answer Result JSON 
Create a Completion Create JSON Snow /advancer! setings [ cance! | 


And lastly there is a Webhook Response with JSON body. Please make sure to 
use Advanced Settings and append "Content-Type" header set to 
"application/json". 


Webhooks Heeb eee 


© Status 

200 

A^ Must be higher than or equal to 100. 
© Body 


© Custom headers Map 


© Item1 = x 


ae o: 


Content-Type I 


A Must be at most 256 characters long 


© Value 


Result JSON Webhooks application/json 


Create JSON Webhook response 


ÀA Must be at most 4096 characters long. 


Add item 


EE Show advanced settings Cancel 


Conversation Summary 


Right now, we don't have any summaries of conversations. Therefore, after a 
response, we can use GPT-3 to generate a summary of the current dialogue. The 
result should be saved in Airtable. 


Nebhooks Summary Save Conversation 


abhook response Create a Completion Create a Record 


As AI Avatar I will summarize for myself, my conversation with 
[name], very precisely marking which words he spoke and which I said. 
44if(41.summary; "I will combine the facts from the summary of the 
previous conversation to create an updated version. 

##H 

Previous summary: 

" + 41.summary) ¢¢ 

#HH 

Latest Dialogue: 

[name] :"4{4{1.content}}" 

Avatar: "{{4{31.choices[].text}}" 
HHH 


Summary of Conversation: 


At this point, we have everything. Our Avatar can both remember and forget; we 
can converse with her, and she will remember our conversation. 


There is one more thing left to do: an interface. It may be anything what can 
send HTTP requests to our webhook, for example Shortcuts app. 


Shortcuts & Siri 


Do you know the Shortcuts app? You can create macros that can be triggered 
with a keystroke or Siri voice command. To interact with our Avatar, we can 
build macro like this: 


GR) AskM.O.N.DAY 


Q 


5) Receive Any inputfrom Quick Actions 


If there’s no input: Categories Apps 


Continue - 
[= All Actions =œ Suggestions 
g : ae 
|] If € Shortcut Input has any value ©? ene Serpthna 
(Sharing Location 
[D Documents Media 
fm Set variable query to £l Shortcut Input ®© Web 


Otherwise 
a) @ i 


® Choose from Menu 


(@ Get clipboard B® Show Alert 


URL 
C-} Ask for Text with What you'd like to know? Show More 


9 
# Show Notification 
fea) 


Show Result 
W Set variable query to $ Provided Input 
Æ Get Variable 


= Stop This Shortcut 
& Enait í Open App 


= Text 


(4) Get contents of https://hook.eul.make.com/ Show Less $ Run Shortcut 
E & o B® Dictionary 


© Get Dictionary from Input 
Method: { POST 5 y R 


e Repeat with Each 
v Headers 


© Get Dictionary Value 
Key Value 


ERER W Get Contents of URL 
Content-Type application/ison 
=) Set Variable 
Show Web View 
@ Base64 Encode 
Open URLs 


® Choose from List 


Request Body: 
Key Value 
content B query Date 
type answer 
© Get URLs from Input 
ri @® Find Reminders 
=) Split Text 


Get dictionary from {@) Contents of URL 


Get Value for message in $ Dictionary 


Copy @ Dictionary Value to clipboard Show More 


As you see, first I get a value from Shortcut Input or from a Clipboard. Then I 
make HTTP POST request to my scenario in make. A response is being parsed 
and copied to the clipboard (and also returned from this Shortcut. It's 


important and I'll show you. why). 

Macros like this can be triggered with a keyboard shortcut on your Mac or with 
a gesture on your iPhone. You can configure it however you want, as you now 
have all the main concepts. 

Voice Interaction 

ElevenLabs is awesome! Based on a 30-60 second audio of your voice, it can 
read any text with it. But the best part, as always, is an API, which you can use in 


Shortcuts app. 


With a very simple macro like this: 


$) Receive Any input from Quick Actions 


If there's no input: 


Continue 


WE Get contents of https://api.elevenlabs.io/v1/text-to-speech/ Show Less 


E EE :/stream 


Method: | POST © 
v Headers 
Key Value 


Content-Type application/json 


Kapika E E oS 


Request Body: | JSON & 
Type Value 


Text £ Shortcut Input 


© Play sound Show More 


You can automatically convert text to audio. Of course, access to this API is paid, 
but it may be more than enough for some interactions. 


Anyway. This macro above won't be triggered directly, but through another 
Shortcut. I have a macro like this: 


Receive input from 


If there's no input: 


Wi Dictate text Show More 


Copy @ to clipboard Show More 


Show More 


Show More 


It's a very simple action that will allow me to dictate text and then trigger both 
previous macros to get a result and read it. 


The fun part is: you can trigger Shortcuts with your voice, using Siri. But this 


time, you'll speak with your AI Avatar! 


Result 


You may be curious as to how this all works. Here is my conversation with 
M.O.N.D.AY.: 


Adam @ 3:03 PM 
What is Inversion (Mental Model)? 


M.O.N.D.A.Y. 8:03 PM 

Inversion (Mental Model) is a mental model that encourages you to 
think about a problem or situation from the opposite perspective. It 
involves flipping the direction of your thinking, and considering how 
to achieve the opposite of your intended goal. It can be used to come 
up with creative solutions, and to identify and address potential 
problems. 


Adam @ 38:06 PM 


Thank you! Do you know other mental models, similar to this? 


M.O.N.D.A-Y. 8:07 PM 

Sure! There are many mental models that are similar to Inversion. For 
example, the First Principles model which consists of breaking a 
problem down into its most fundamental parts and seeing which 
solutions can be derived from them. Another model is the 80/20 
principle which states that 80% of the effects come from 20% of the 
causes. There are also models like the Pareto Principle, Occam's 
Razor, and the Ripple Effect that can be used to help solve complex 


problems. 


I can communicate with her using both Siri Shortcuts and Slack. As I said, you 
can use any interface you want; all you need to do is send a simple HTTP 
request. 


On the screenshot above, you can clearly see that she can answer my questions 
and is aware of the context of our conversation. In the second message, I just 
said, "similar to this," and she was aware of what I was asking. 


In fact, the possibilities are almost endless here. I'm pretty sure that you'll need 
to make some changes to the examples of prompts I gave, but still, you've got a 
core idea, so you can use it. 


Summary 


Creating an AI Avatar using no-code tools may seem complicated, but 


everything you see in this article was created by me in a couple of days. There's 
no doubt that this approach could be better. Right now we're using only "in- 


prompt learning", which is slow and cost more. But also it's way more flexible 
than fine-tuning for example. Moreover, M.O.N.D.A.Y. is far more advanced than 
what you can see here. I will definitely share my future experiences, ideas, and 
techniques; 


Just follow my profile. 


